I just realized that saving the document may lead to a failure to run the tests if the name of a describe or test is changed. I'll close this while I work on a better solution.

Reopening this because handling renamed tests ended up being quite a bit more complicated than I had hoped.

From what I could find, most other extensions that implement similar test running through code lenses in VSCode do it through the CodeLensProvider, which appears to be called on code change (meaning the lenses are effectively always up to date), whereas the LSP command appears to only be called on document save. Obviously, this behaviour may differ between editors.

One avenue to handle this is to re-request code lenses when we detect that the file is dirty, but then the problem becomes matching those new lenses with the current state of the document. Considering that tests can be renamed, moved around, moved between describes, etc., this feels very hard to do accurately, if possible at all.

Another option is the Code Lens Refresh Request defined by LSP. This would have to be implemented in elixir_ls and would allow us to refresh code lenses in-between saves, but refreshing on change feels like a bad idea given the time it currently takes to generate lenses. There's always the option of implementing some kind of debouncing, but the cost/value ratio is starting to drop dramatically imo.

@axelson Sorry for tagging you out of nowhere but I wasn't sure anyone would look at this since it's been stagnant for a month and a half.

The TL;DR of my last comment is that I think this is the best we can do without investing significant effort into this issue, which I don't think is worth it...

So I've pulled master and I think this is ready to be merged 🙂